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Summary 

Objectives Health risk assessment (HRA) questionnaires have become 
a popular tool to help quantify health issues within populations. Over the 
last decade HRAs have increasingly been delivered in the online 
environment. The objective of this study was to create and validate an HRA 
that is optimized for delivery via the Internet. 

Design After an iterative process of user testing and interface design 
the RedBrick Health Assessment (RBHA) was validated against known 
domain specific questionnaires with 464 working Americans, and with 
medical claims data from over 25,000 employees. 

Setting All consumer testing, data capture and analysis occurred at the 
offices of RedBrick Health Corporation, Minneapolis, USA and via a secure 
online portal. 

Participants Individuals in full-time employment in the USA, who 
were between 18 and 65 years of age at the time inquiry. 

Main outcome measures Correlation of the included RBHA 
domains with the output from known gold standard health question sets 
for each assessed health domain. 

Results The iterative development process employed in creating the 
RBHA produced a tool that had a high degree of user acceptability. The 
domains demonstrated good correlations with relevant gold standard 
questionnaire measures, good internal consistency, and acceptable 
sensitivity and specificity when compared to gold standard risk 
stratification and high-risk classification (specificity of domains ranged 
from 76-94%). Atest-retest correlation co-efficient of 0.7, or greater, was 
achieved 8 weeks after initial completion. 

Conclusions The RBHA is a new breed of HRA that has been 
specifically developed for capturing health status information in an online 
environment. At its heart is user centricity and this focus has enabled the 



J R Soc Med Sh Rep 2011 2:71. DOI 10.1258/ shorts.201 1.01 1015 



J 



Journal of the Royal Society of Medicine Short Reports 



Ethical approval 

Formal ethical 
approval was not 
sought for this study 
as it was conducted 
outside of a hospital 
or academic 
institution. However, 
all participants were 
required to accept 
privacy and security 
terms that were 
specific about how 
their data were to be 
used, who had 
access to the data, 
and where the data 
would be stored. All 
data were stored and 
protected in 
accordance with the 
Health Insurance 
Portability and 
Accountability Act 
(HIPPA) 1996. 
Individuals were 
explicitly informed 
that they were free 
to cease 
participation in the 
study at any stage 
during its course and 
have their data 
deleted from our 
database 

Guarantor 

PRM 

Contributorship 

PRM, WSM, CMB, 
MEH and JRWB 
developed and 
designed the RBHA; 
PRM and WSM 
designed the 
investigative 
protocol; WSM and 



creation of a tool that is not only highly engaging but also captures 
accurate and robust health status information. 



Background 

Over the last decade health risk assessment (HRA) 
questionnaires have become increasingly popular 
tools used by organizations and health insurers 
to help quantify the health issues of their 
employee or member base. 1 A recent survey 
found that 74% of US and 46% of UK companies 
offer an annual HRA to their employees as part 
of their employee health management initiatives. 
The popularity of this method of data capture is 
likely to increase in the coming years as more 
organizations accept that maintaining a healthy 
workforce can deliver real productivity and per- 
formance benefits. 2 

HRAs typically capture responses on a wide 
range of medical, lifestyle and behavioral topics. 
In addition some also inquire about biometric 
and quality of life parameters in order to create a 
broad overview of an individual's health status. 
A review of the historical background, scientific 
basis and uses of HRAs is beyond the scope of 
this paper, however the reader is referred to two 
good overview chapters in the most recent 
edition of the American College of Sports Medicine's 
Worksite Health Handbook for context. 3,4 

HRA data are valuable in a number of distinct 
ways. In their amalgamated form they can help 
organizations and healthcare providers target 
appropriate resources for individual and group- 
wide interventions. 5 In addition, together with 
historical medical and pharmacy cost data, they 
can also help predict future cost liabilities, both 
in terms of medical care requirements, but also 
in relation to work performance and pro- 
ductivity. 2 ' 6-8 From the individual perspective 
they can help respondents better understand 
their own health status, how they compare to 
others like them, as well as the specific areas 
that they should focus on improving. It is unli- 
kely, however, that this knowledge alone will 
effect significant population-wide behavior 
change. Rather HRA completion needs to be aug- 
mented with appropriate consumer-focused 
health improvement programs to achieve health 
benefits. 3 



The last 10 to 15 years has seen a wealth of 
research on predicting costs of medical care, 
absenteeism, productivity and disability. 9 - 15 In 
the majority of these studies data from an HRA 
were an essential first step in defining the 
'at-risk' population and observing how specific 
health risks relate to the different cost parameters. 

With the increasing penetration of the Internet 
in both business and society as a whole, HRAs 
have shifted from being a predominantly paper- 
based solution to being almost exclusively deliv- 
ered via the world wide web. 16 The advantages 
of this approach are significant; web delivery 
allows for a lower cost of implementation as well 
as almost limitless scalability compared to paper 
implementation. These practical considerations, 
together with the interactivity and ability of the 
web to serve up 'real-time' information and rec- 
ommendations to consumers, make it the main 
delivery channel for the majority of vendors and 
purchasers of HRA services. 

Over 75% of the US population has access to 
the Internet, with the vast majority of these indi- 
viduals having broadband access. 17 Outside of 
the US a number of countries in Europe and 
Asia have even higher population penetration 
of Internet services. With this almost ubiquitous 
Internet presence in our lives it is perhaps sur- 
prising that there has been so little published 
work examining how to optimize the HRA 
experience for online delivery. Many commer- 
cially available HRA products started out as 
paper tools that subsequently got placed into 
online environments, with little or no modifi- 
cations to layout or emphasis. In addition, 
despite their extensive use there are very few 
that have gone through a rigorous questionnaire 
validation process. 5 

Although evidence does suggest that responses 
are not significantly different when comparing 
completers of paper and online versions of the 
same questionnaires there would appear to be 
some utility in creating a better, more robust and 
validated online HRA that makes use of the 
unique characteristics of the 21st century world 
wide web. 18 
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Recently the Patient Protection and Affordable 
Care Act in the United States has mandated that 
an HRA needs to be part of the annual wellness 
visit Medicare beneficiaries are entitled to. Simi- 
larly, healthcare reform in the UK is placing 
much greater emphasis on preventive health strat- 
egies as a way to mitigate increasing healthcare 
delivery costs. 

In this article we report on the design, develop- 
ment and validation of a next generation HRA. 
The RedBrick Health Assessment (RBHA) has 
been created specifically for the online environ- 
ment so as to maximize engagement, understand- 
ing and accuracy of completion as well as 
minimize respondent time commitment necessary 
to achieve this. 



Methods 

The development of the RBHA involved four sep- 
arate phases: (1) design of the graphical user inter- 
face; (2) development of the question set; (3) user 
testing and face validation; and (4) validation of 
the included domains against domain specific 
'gold standard' measures. 



Design of the graphical user interface 

The graphical user interface (GUI) design process 
started with a review of existing, easily accessible, 
online HRAs and the approaches taken to ask 
questions and collect information. This review 
demonstrated an almost exclusive use of text in 
question stems and answer options. The use of 
scripting to surface relevant information or to 
skip or re-word irrelevant questioning, based 
upon previous answers, was used sparingly and 
by a minority (less than 40%) of the HRAs that 
we reviewed. In general it was concluded that 
the commonly available HRAs were text heavy, 
lacking in graphical components that could aid 
the users' understanding, and took a significant 
amount of time to complete. 

Based upon this initial market scan we devel- 
oped a number of templates that could be popu- 
lated with specific health-related questions. To 
aid this process we reviewed some of the most 
popular, non-health-related websites, including 
Amazon, eBay and Expedia to observe how they 
presented information and asked users questions. 



The templates incorporated placeholders for 
images as well as interactive response options. 
Figure 1 gives examples of some of these 
templates. 

Prior to populating the templates with specific 
health-related questions they were all tested on a 
group of six volunteers to ensure usability and 
ease of understanding. Dummy questions were 
used while volunteers were video-taped complet- 
ing them on screen. Software that tracked eye 
movements as well as mouse movement and 
clicks (Silverback 2.0, Brighton, UK) was also 
employed. 

The output from this initial user testing 
resulted in minor changes to the GUI prior to 
population with the health-specific questions. 



Development of the question set 

Prior to the development of the question set we 
reviewed the scientific literature to establish 
which health-related domains to create questions 
for. Based on the published evidence for specific 
health, wellness and lifestyle areas having an 
impact upon medical and pharmacy claims 
costs, absenteeism, productivity, workers compen- 
sation or disability claims, and also the previous 
research that one of the authors (PRM) has done 
in the area, we developed questions for 15 separate 
domains in the RBHA. 5 

Using the templates developed in phase 1 an 
iterative process of question development and 
user testing was employed until we were satisfied 
that each question was easily understandable by 
users and collected the necessary information 
required to stratify respondents into risk groups 
for each of the domains. 



User testing and face validity 

Once the whole question set had been developed, 
end-to-end testing was undertaken with a cohort 
of 1200 employees from a national employer. Par- 
ticular attention was paid to the length of time it 
took individuals to complete the RBHA as well 
as the proportion of the population classified as 
high-risk in each of the domains. In addition 
each respondent was asked to provide feedback 
on ease of use as well as areas that could be 
improved. 
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Figure 1 ' 

Examples of interactive response options utilized in the RBHA 
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In addition to the end-user testing of the RBHA 
we also engaged a number of industry experts to 
provide their opinions and recommendations for 
improvements. Three individuals, each with at 
least a decade of experience in research and devel- 
opment of population health management sol- 
utions, including the development of HRAs for 
health plans and commercial health management 
organizations, were provided with online access 
to the HRA and asked for their general comments 
as well as responses to a short structured 
questionnaire. 

This phase of testing resulted in changes to 
templates, layouts and question wording in 
order to improve accuracy and timeliness of 
completion. 

Validation of the RBHA 

The final phase of the development of the RBHA 
was validation against relevant 'gold standard' 
measures for the included domains. Although 
there is no universally recognized 'gold standard' 
HRA there are a number of well validated 



questionnaires that assess many of the domains 
included within the RBHA. 7,19-23 The purpose of 
this part of the development process was to 
ensure that the questions that required non-binary 
responses correlated well with the chosen gold 
standard measure of that domain. A number of 
appropriate gold standard question sets were 
identified and incorporated into our online 
environment for the purposes of this part of devel- 
opment process. The gold standard question set 
was randomly positioned before or after the 
RBHA so as to minimize any 'training' impact 
one question set may have on the other based 
upon which was answered first. Table 1 provides 
further details on the gold standard measures 
used. 

In addition to correlating responses to domain 
questions between the RBHA and the gold stan- 
dard questions sets, those domains that were com- 
posed of multiple items were also examined for 
internal consistency by calculating the Cronbach-a 
and inter-item correlation values. 

Volunteers were recruited by an online survey 
organization (Market Tools Inc, San Francisco, 



Table 1 ^ 






Gold standard questionnaires used in the validation phase of the RBHA development 


Gold standard questionnaire 


Corresponding 


Description of questionnaire 




domain in RBHA 




12-item General Health Questionnaire 


Stress 


12 questions related to mental health 


(GHQ-12) 22 




symptoms over preceding 'few' weeks 


Short Form Rapid Eating and Activity 


Nutrition 


13 questions related to general food 


Assessment for Participants 




choices and preferences 


(REAP-S) 20 ' 21 






Pittsburgh Sleep Quality Index 


Sleep 


9 questions examining sleep time and 


(PSQI) 19 




sleep disrupters over the preceding 






month 


Short Form International Physical 


Physical activity 


4 questions examining amount of vigorous 


Activity Questionnaire (IPAQ) 23 




and moderate physical activity and the 






amount of walking an individual does 






over 7 days 


Short Form 36 Health Survey (SF-36) 


General health 


36 questions examining health-related 




Life satisfaction 


quality of life over the preceding 4 weeks. 




Pain 


For the purposes of the RBHA validation 






study only 3 questions from the original 






questionnaire were used 


Part B of World Health Organization 


Productivity 


12 questions that start with priming 


Health and Work Performance 




questions to get the respondent thinking 


Questionnaire (WHO-HPQ) 7 




about their role and end with getting to 






individual to rate their productivity over 






the preceding 4 weeks 



J R Soc Med Sh Rep 2011 2:71. DOI I0.1258/shorts.20l1.011015 5 



Journal of the Royal Society of Medicine Short Reports 



CA, USA) to complete both the RBHA and the 
appended gold standard questions in one sitting. 
Individuals were incentivized to complete the 
question set by the survey organization with a 
$25 voucher redeemable at a number of online 
outlets. Eligibility requirements were: (1) being 
in full-time employment within the United 
States of America; (2) being 18 to 65 years of age 
at the time of survey completion; and (3) having 
access to the Internet. All respondents were 
asked to complete the full question set within a 
three-week period during July 2010. A random 
subgroup of 100 respondents were asked to com- 
plete the RBHA eight weeks after their initial com- 
pletion date in order to assess test-retest validity 
of the questionnaire. 

The final part of this phase of the investigation 
validated respondents' answers to the medical 
condition checklist with historical medical and 
pharmacy claims data. Data were merged from a 
cohort of 27,215 individuals from a variety of 
different employers who completed the RBHA in 
the latter half of 2010 and who also had claims 
data for the period 2009-2010. Using medical 
and pharmacy claims as the 'gold standard' (Sym- 
metry Episode Treatment Groups, Ingenix, USA) 
we analysed the sensitivity and specificity of the 
self-reported medical conditions question of the 
RBHA for correctly identifying common 
conditions. 

All data analysis was carried out using Statis- 
tica 9, a statistical software package distributed 
by Statsoft, Inc (Tulsa, OK, USA; http://www. 
statsoft.com). 

Results 

The user-centric development process employed 
to create the RBHA enabled us to develop a new 
level of insight into how individuals interact 
with onscreen HRA questions, as well as how to 
present questions and answers so that responses 
are both intuitive and accurate. 

Figure 2 shows an example of the GUI iter- 
ations that were developed as a result of user feed- 
back and the way individuals responded to the 
questions in the RBHA. At each step in the devel- 
opment process user feedback allowed us to hone 
the graphics, wording and layout of the questions 
so that when presented on screen the respondent 



quickly understood what was being asked of 
them, as well as how to answer appropriately. 

Face-validity testing with industry experts 
allowed us to get an unbiased overview of how 
the RBHA compared to other tools in the market 
place. All three experts agreed that the RBHA 
was a significant improvement on existing HRAs 
as it provided the respondent with a more intui- 
tive and easy to understand interface than cur- 
rently exists. In addition, all three experts 
strongly agreed that the included domains and 
the way the questions within the domains were 
presented were appropriate and valid. 

Table 2 provides further detail on the RBHA 
domains and the types of question included 
within each domain. 

Six hundred individuals were invited to partici- 
pate in the external validation phase of the RBHA, 
of these 464 (77%) completed both question sets 
within the three- week timeframe. The mean age 
of respondents was 43.3 years (SD 14.4), 54% 
were women and 86% classified themselves as 
Caucasian. 

The mean time respondents took to complete 
the RBHA plus the gold standard question set 
was 35.8 min (median 32 min). The mean time to 
complete the RBHA alone was 29.5 min (median 
18.2 min). 

Table 3 shows how eight of the RBHA domains 
correlated with their associated gold standard 
measure. For each domain a Pearson correlation 
co-efficient (r value) between the responses to 
the relevant RBHA questions and those of the 
gold standard was established. In addition, for 
each gold standard questionnaire a cut-off value 
commonly used to indicate 'high-risk' or 'at-risk' 
status was used to calculate the sensitivity, speci- 
ficity, and positive and negative predictive 
values (PPV and NPV) of the RBH domain. 

As well as looking at the correlation 
between overall nutritional balance by comparing 
the computed score from the six nutritionally- 
focused questions in the RBHA with the 
REAP-S questionnaire, we also examined the 
observed correlations between the major constitu- 
ent food groups. Statistically significant (P < 0.001 
for all) correlations were shown between the 
RBHA measures of fat intake, fibre intake, salt 
intake, and fruit and vegetable consumption 
with the appropriate question sets within the 
REAP-S. 



6 J R Soc Med Sh Rep 2011 ;2:71. DOI 10.1258/ shorts.2011 .011015 



Development of new online health risk questionnaire 



^Figure 2 » 

Four screen shots showing evolution of the physical activity question as a result of iterative user feedback 

Iteration one: 




How do you spend your time on a typical week day? 

Please Indicate Ihe amount ol time you spend at each activity level (24 hours total) 



You are 8% complete. 



Sleeping/Lying Down 

(lying in pad, wauling TV. ttc) 

Sitting 

(at your d«k tating. Matching 



man mopping. 



Standing 

(washing dilhti. 

Walking 

(100. playing won kids pelt) 



jogging 

(bicycling, yard work) 

Running 

(MfOOKV wttgiM lifting. ro**ng> 



05 





Movt It To Lose It 



Phvs<*lh/ ft people burn more 
calories 4i rest than those who 
Arc out of Shape simply by 
having more muscles Each 
pound of muicJe <) pound - 
0 45 kilograms) burns 75 to 
1 00 ealones every day Simply 
by being 



24 



Iteration two: 




How do you spend your time in a typical day? 

Input the amount ot lime you spend (in hours) doing each activity during a normal week day 
and weekend or off day Remember this is the amount ol time per day and not the total amount 
during the whole week 



You are 8% complete. 




Day-to-day Activities 

Level 1: Sleeping & Lying Down 

Relaxing Sunbathing Sleeping Watching TV Working 

Level 2: Sitting Down 

Watching TV Eating Driving Working at Desk Meetings 

Level 3: Standing, walking around, housework and other similar activities 
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Figure 2 
Continued 

Iteration three: 



Do you exercise? 



6 Yes 

O No 

Indicate the different types ot exercise you get during a typical week 



You are 8% complete. 




0 Moderate Excercioe 



How long? 0 
How often? 0 



i I minutes 



M times per week 



_ Vigorous Exercise 

Activity that pushes you physically, making 
you breathless and sweaty. 



_ Extremely Vigorous Exercise 

Intense activity like competitive sports 
performed al a high level. 



Activity that makes you breathe harder 
and leel warm. 





« Previous Question 



Next Question ■ 



Final: 



Do you exercise? 



Exercise doesn't always mean going to the gym. II you do 
gardening, you're gelling exercise. 



9 Yes 

O No 

Moderate Excerclse 

Activities that make you breathe harder and feel warm 



aclrvrbes such as brisk walking and 



You are 8% complete. 




Vigorous Exercise 

Intense activities that make you feel out of breath and sweat 





Think about a typical week How much moderate exercise do you get? 

Sunday j o Us) hours 1 n ««x) minutes 

Monday 0 *j hours ; o jtjj minutes 

Tuesday \ 0 isjj hours 0 >H iis n u m 

Wednesday 0 H hours > o mj minutes 

Thursday [oM hours (JM§ minutes 

Friday ! 0 »H hours 1 0 jjsj minuies 

Saturday ! cT hours GO 



Think about a 

Sunday 

Monday 

Tuesday 

Wednesday 

Thursday 

Friday 

Saturday 



typical week. How much vigorous exercise do you get? 

0 iH hours Q m minutes 

0 hours Q tit) minuies 

[ 0 Hi hours : 0 fcssj minutes 

10 its) hours ; 0 itx) minutes 

; 0 kssj hours 0 •») minutes 

; 0 HBj hours j 0 ess) minutes 

! 0 *H hours ; 0 k»j minutes 
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Table 2 a 








Details of the 15 domains in the RBHA and an overview of how the responses to the questions within the 


domains were used to attribute risk status 






RBHA domain Questions, 


Question areas of focus 


Scoring and high-risk 


Validated 


items or data 




classification 


against a 


nrtint<z that 






'gold 


constitute the 






standard' - 


domain (r\) 






Y/N 


( o i-fl i t~\\/acr , i i 1 Qr 
odlUIUVdbOUIdl O 


Pnmm itatinn /~»f 1 D.\/QQr 
UUI 1 1 |JU LdUU 1 1 Ul 1 U ycdl 


Scoring based upon 


N 


risk 


cardiovascular risk 


Framingham relative 






ubi i ly ridiiMiiyiidiii 


risk calculation for 






cLjUdLIUIIb dllU Ubci 


cardiovascular event in 






inputted data on sqg, 


the next 10 years. 






LULdl OllUlcbLclUI, nUL 


High-risk status denoted 






ON U 1 cbLol U 1 , UIUUU 


by Framingham relative 






pressure, diabetes and 


risk of >3.5 






b 1 1 1 u i\ 1 1 ) y 






IMUTrlTIOn D 


Separate anchored visual 


Choices for fat, fibre, salt. 


Y 




dlldiuy bCdlc LjUcbLIUIIb 


and fruit and vegetable 






on usual fat, fibre, salt, 


consumption given 






dllU IIUIL dllU VcycLdUlc 


equal weighting. 






consumption. 


Responses to snacking 






Additionally, questions 


and beverage questions 






on usual snacking 


amalgamated and given 






1 1 d u I Lo d 1 1 u 


same weighting as one 






non-alcoholic 


of the items above. 






beverages 


Overall nutrition score 








computed on scale of 0- 








100 with score of <50 








denoting high-risk 








status 




DOUy WciyilL O 


LUl 1 1 pu IdllUI 1 Ul uuuy 


Scoring according to 


N 




[lldbb IIIUca \DIVII/ [[(Jill 


NHLBI recommended 






user-inputted data on 


ranges for BMI. 






height and weight. 


High-risk denoted by 






VvdlbL U 1 [ OU 1 [ 1 1 c [ cl 1 L/C 


BMI >30 and/or waist 






used for additional risk 


circumference >40 in for 






weighting 


men and >35 in for 








women 




Tobacco use 1 


Current and past tobacco 


Current smokers (or users 


N 




usage 


of other tobacco 








products) classified as 








high-risk 




Alcohol 1 


Computation of total 


Respondents exceeding 


N 


consumption 


weekly alcoholic drink 


national guidelines for 






consumption 


safe alcohol 








consumption classified 








as high-risk 




Sleep 3 


Average sleep hours per 


Average nightly sleep 


Y 




day computed from 


hours summed with a 






question on usual 


factor of 1 -5 based 






weekday and weekend 


upon how respondent 






sleep hours. Overall 


feels an hour after 




(Continued) 
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Table 2 v 










Continued 










RBHA domain 


Questions, 


Question areas of focus 


Scoring and high-risk 


Validated 




items or data 




classification 


against a 




points that 






'gold 




constitute the 






standard' - 




domain (n) 






Y/N 






feelings of vitality soon 


waking. Value of <9 








after waking in the 


classified as high-risk 








morning. 






Stress 


2 


Self-reported level of 


Stress 0-10 score 


Y 






stress on 0-10 visual 


multiplied by a factor 








analog scale. 


between 0.75 and 2 








Individual's ability to 


depending upon how 








cope with the current 


respondent states they 








level of stress in their 


are coping. Value of >8 








life 


classified as high-risk 




Risk behaviour 


1 


Checklist of eight risks an 


Equal weight given to all 


N 






individual may take in 


eight risks. Individuals 








their day-to-day life, 


with >3 risks classified 








including seatbelt 


as high-risk 








usage, helmet usage, 










drinking and driving, 










safe storage of firearms 










and use of sunblock 






Pain 


1 


Multi-part question 


Each respondent scored 


Y 






asking individual to 


on 0-4 scale for pain at 








indicate the pain they 


the seven sites. All 








currently experience 


scores summed. High 








from seven bodily 


risk denoted by score of 








locations and then rate 


>11 








each site of pain on a 










5-point Likert scale from 










mild to very severe 






Physical activity 


1 


Computation of MET 


Respondents with <750 


Y 






min/week derived from 


MET min/ week of 








moderate intensity and 


activity classified as 








high intensity physical 


high-risk 








activity 






Medical health 


1 


Checklist of 13 common 


Respondents with 3 or 


N 






medical conditions 


more medical 










conditions classified as 










high-risk 




General health 


1 


5-point Likert scale asking 


Respondents stating their 


Y 






respondent to rate their 


health is 'poor' or 'fair' 








overall health status, 


classified as high-risk 








ranging from Poor to 










Excellent 






Life satisfaction 


1 


5-point Likert scale asking 


Respondents stating their 


Y 






respondent to rate their 


life is 'terrible' or 








overall life satisfaction. 


'difficult' classified as 








ranging from 'terrible' 


high-risk 








to 'great' 






(Continued) 
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Table 2 i 










Continued 










RBHA domain 


Questions, 


Question areas of focus 


Scoring and high-risk 


Validated 




items or data 




classification 


against a 




points that 






'gold 




constitute the 






standard' - 




domain (n) 






Y/N 


Preventive care 


1 


Checklist of age- and 


Respondents scored 


N 






gender-appropriate 


according to the 








preventive care services 


proportion of 








recommended by the 


recommended 








US Preventive Services 


preventive services they 








Task Force 


have had in the 










appropriate timeframe 




Productivity at 


2 


0-10 anchored visual 


Respondents rating their 


Y 


work 




analog scale for 


productivity as <6 on the 








self-rated productivity. 


0-10 scale classified as 








Question on number of 


high-risk 








work days missed due 










to ill-health in previous 










3 months 







Examination of the internal consistency of the 
six items within the nutrition domain of the 
RBHA revealed a Cronbach-a value of 0.74 and 
an inter-item correlation of 0.37. 

One hundred randomly selected individuals 
were invited to retake the RBHA eight weeks 
after they had originally taken the questionnaire. 
Eighty-two responded, the average time between 
original completion and repeat completion was 
61.4 days (range 50.3-67.6 days). For all of the 
domains the Pearson correlation co-efficient was 
0.7 or higher (P < 0.001 for all). 

The sensitivity and specificity of the RBHA 
medical checklist items in correctly identifying 
medical conditions flagged by a medical and 
pharmacy claims analysis program ranged from 
65-75% and 72-94%, respectively. 

Discussion 

HRAs, and the data they generate, have become an 
integral component of population health manage- 
ment strategy for many employers, health insurers 
and healthcare providers. 1 The last decade has 
seen HRA deployment shift from being predomi- 
nantly paper-based to almost exclusive delivery 
via the Internet. With this shift in delivery 



channel comes a need to ensure the tools that we 
are using, together with how they are deployed, 
are valid and optimized in order to provide the 
best possible foundation for subsequent popu- 
lation health interventions. 

The characteristics of the online environment 
are markedly different from print media, and 
although a web page can function as a digital rep- 
resentation of a piece of paper there is now the 
opportunity to use modern browser functionality 
to enhance user experience and accuracy of data 
collection. The RBHA represents such a new 
breed of HRA in that it has been developed 
specifically for Internet delivery and undergone 
extensive user testing and validation analyses. 

In developing the RBHA we looked outside the 
traditional health and medical paradigms that we 
normally operate within and from the outset 
employed recognized principles of good website 
development and design that are common in the 
retail and marketing sectors. Although the 
process added a significant amount of time to 
the HRA development it did enable us to create 
a tool that has user-centricity at its core and that 
we believe is unique in the market place. 

One of the central tenets of the development 
process for the RBHA was to keep completion 
time as short as possible. Our own experience of 
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Table 3 ^ 

Table showing the Pearson correlation co-efficients (r values) for the relationship between the measured RBHA domain and 
the associated gold standard question set. For each gold standard measure a commonly used score or response 'cut-off 
value was used to classify respondents as high-risk. These cut-off values were used to assess the sensitivity, specificity, 
negative predictive value (NPV) and positive predicative value (PPV) of the RBHA domain question(s) as compared to the 
gold standard questionnaire 



Domain 



Gold 

standard 

measure 



Gold standard 
high-risk 
cut-off value 



Correlation 
co-efficient 
(r value) 



Population 
classified as 
high-risk by 
gold standard 

(%) 



Population 
classified as 
high-risk by 
RBHA (%) 



Sensitivity and 
specificity 



PPV/ NPV 



Stress 

Nutrition 

Sleep 

Physical 

activity 
General 

health 

Life 

satisfaction 



Pain 



Productivity 



GHQ-12 
REAP-S 
PSQI 

IPAQ Short 

Form 
SF-36 (Pain 

question) 

SF-36 
(Physical / 
Emotional 
problems 
question) 

SF-36 (Pain 
question) 



WHO-HPQ 
(Part B) 



Score >3 
Score <23 
Score >7 

<750 MET 

min/week 
Response of 

'Poor' or 

'Fair' 
Response of 

'All of the 

time' or 

'Most of the 

time' 
Response of 

'Extremely' 

or 'Quite a 

bit' 
Score of <6 



r = 0.60* 
r = 0.67* 
r = 0.55* 
r = 0.56* 
r = 0.50* 

r = 0.54* 

r = 0.54* 
r = 0.58* 



16 
29 
25 
62 
14 



12 



22 
32 
24 
47 
19 



11 



Sensitivity: 59% 
Specificity: 85% 
Sensitivity: 71% 
Specificity: 83% 
Sensitivity: 54% 
Specificity: 87% 
Sensitivity: 60% 
Specificity: 76% 
Sensitivity: 61% 
Specificity: 88% 



PPV: 
NPV: 
PPV: 
NPV: 
PPV: 
NPV: 
PPV: 
NPV: 
PPV: 
NPV: 



0.43 
0.92 
0.63 
0.87 
0.57 
0.85 
0.80 
0.54 
0.45 
0.93 



Sensitivity: 40% PPV: 0.37 
Specificity: 93% NPV: 0.93 



Sensitivity: 40% PPV: 0.27 
Specificity: 94% NPV: 0.96 



Sensitivity: 48% PPV:0.39 
Specificity: 92% NPV: 0.94 



*P< 0.001 

GHQ-12 = General Health Questionnaire 12; REAP-S = Short Form Rapid Eating and Activity Assessment; PSQI = Pittsburgh 
Sleep Quality Index; IPAQ = International Physical Activity Questionnaire; SF-36 = Short Form 36 Questionnaire; 
WHO-HPQ = World Health Organization Health and Work Productivity Questionnaire 



implementing HRAs has shown significant 
drop-off in completion after 20 minutes of engage- 
ment, we therefore worked to making sure com- 
pletion time did not exceed this. To this end the 
majority of the domains that make up the RBHA 
consist of single or double items, with only nutri- 
tion having a greater number of questions. This 
approach kept the necessary completion time to 
a minimum, but as we have shown, still enabled 
the collection of reliable health data. 

At the outset it should be recognized that an 
HRA is not a 'diagnostic' tool, rather it is an instru- 
ment that can assist in categorizing individuals 



into risk groups based upon their responses, 
much like triage in the medical world. In essence 
an HRA is triaging respondents so that those indi- 
viduals considered to be at increased risk in 
specific areas can receive further assessment and 
intervention as necessary. 

As well as streamlining the user experience we 
also wanted to make sure that the data we were 
collecting were robust and 'measured what we 
thought we were measuring'. There are very few 
formal HRA validation studies published in the 
scientific literature, which in itself begs the ques- 
tion as to whether some instruments are accurately 
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collecting health risk data. By comparing the main 
RBHA domains with simultaneously collected 
responses from gold standard domain specific 
questionnaires we have demonstrated statistically 
significant correlations in all areas. This suggests 
that at a minimum the RBHA domains are 
tapping into the same constructs as the gold stan- 
dard question sets. In addition, we chose com- 
monly used cut-off values for each of the gold 
standard questionnaires to represent the boundary 
between high-risk status and normality in each of 
the investigated domains. Clearly using such cat- 
egorization is not diagnostic, but in the absence 
of formal physician review of each respondent it 
gave us an indication of how the RBHA might 
perform in such situations. 

As is the case with many questionnaires, 
especially those that use a few items in each 
domain, the sensitivity of the RBHA questions in 
each domain was modest; however for the main 
part the specificities and negative predictive 
values (NPV) were high. Having a high value for 
specificity and NPV is an important attribute of a 
screening/triage questionnaire as one can be 
reasonably confident that if an individual is classi- 
fied as not being at risk this will indeed turn out to 
be the case. As an HRA should be used as a triage 
instrument those individuals who are classified as 
high risk will inevitably go through further 
inquiry as to whether they meet risk criteria or 
not. The only area where the RBHA specificity 
fell below 80% was in the physical activity 
domain. This may be due to the fact that the 
IPAQ categorized more than 60% of the popu- 
lation as not meeting minimum levels of regular 
activity; a figure that is significantly higher than 
published epidemiological data for the United 
States. 24 

Similarly, medical and pharmacy claims data 
are not always accurately coded, however, 
despite this potential limitation respondents 
answers' to the question enquiring about current 
medical diagnoses showed a high degree of corre- 
lation with diagnoses derived from claims. 

Not all of the domains in the RBHA were 
included in the gold standard validation process 
reported in this paper. There were a number of 
reasons for this; for some domains (cardiovascular 
risk, body weight, preventive care) standard algor- 
ithms or consensus recommendations were used, 
and unchanged, to classify risk status. For other 



areas (smoking, alcohol consumption) respon- 
dents were simply asked to quantify their usage 
status, much as a physician would enquire and 
record during a medical history. In addition, for 
some areas we could not find an easy-to-adminis- 
ter questionnaire that covered the area of the 
domain in question (risk behavior). 

The presented research is only the start of 
ongoing validation work for the RBHA. Further 
research will look specifically at the ability of the 
RBHA to prospectively identify individuals who 
will have high healthcare utilization and high 
cost interactions with the healthcare system in 
the short to medium term. 

Conclusions 

In summary, we present the process by which we 
have created a new health risk assessment instru- 
ment. With the increasing digitization of almost all 
aspects of our lives, we believe it is imperative that 
the creation of consumer-focused health-related 
tools and applications are grounded in consumer 
usability and user centricity as well as scientific 
validity. 
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